Exercise 3

Let us denote \[\begin{align*} \mathbb{C} &= \begin{pmatrix} \mathbb{A} \\ \mathbb{B} \end{pmatrix}. \end{align*}\] It follows that \[\begin{align*} \begin{pmatrix} \mathbb{A}\mathbf{X} \\ \mathbb{B}\mathbf{X} \end{pmatrix} = \begin{pmatrix} \mathbb{A} \\ \mathbb{B} \end{pmatrix} \mathbf{X} = \mathbb{C} \mathbf{X}. \end{align*}\] Since linear transformation preserves normality, we know that \[\begin{pmatrix} \mathbb{A}\mathbf{X} \\ \mathbb{B}\mathbf{X} \end{pmatrix} \sim \mathcal{N}_2 \left( \mathbb{C}\mu, \; \mathbb{C}\Sigma \mathbb{C}^T \right). \] We observe that \[ \mathbb{C}\Sigma \mathbb{C}^T = \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} = \begin{pmatrix} 2 & 0 \\ 0 & 2 \end{pmatrix}. \] Therefore \(\mathbb{A}\mathbf{X}\) and \(\mathbb{B}\mathbf{X}\) are uncorrelated. Since they are jointly gaussian, zero correlation implies independence.

Let us proceed to the empirical evidence of our claim. Let us generate a sample from \(\mathcal{N}_2 \left( \mu, \; \Sigma \right)\) and transform it by \(\mathbb{A}\) and \(\mathbb{B}\). Firstly we will check empirically whether it is true, that they are jointly gaussian. Hence we will compare contours of their estimated density with contours of \(\mathcal{N}_2 \left( \mathbb{C}\mu, \; \mathbb{C}\Sigma \mathbb{C}^T \right)\) density.

set.seed(2112)
n = 10000
X = rmvnorm(n, c(2, 2), matrix(c(1, 0, 0, 1),2,2))
A = t(matrix(c(1,1)))
B = t(matrix(c(1,-1)))
C = rbind(A,B)
U = t(A%*%t(X))
V = t(B%*%t(X))

den3d <- kde2d(U,V)
Sigma = C%*%t(C) ## variance-covariance matrix

x1 = seq(4-4,4+4,0.01)
x2 = seq(0-4,0+4,0.01)
contour(x1,x2,outer(x1,x2,function(x,y){dmvnorm(cbind(x,y),mean = c(4,0),sigma=Sigma)}),
        col = "blue", lty = 1, drawlabels = F, main = 'Density contours')
contour(den3d, col = 'red', add = T, lwd = 2, lty = 2)
legend('topright', legend = c('Theoretical density','Estimated density'),
       col = c('blue','red'), lty = c(1,2), cex=0.8)

The contours are almost indistinguishable, hence there is some evidence that jointly they follow gaussian distribution.

Now we will investigate the correlation. The shape of the contours already suggest very low correlation. Sample correlation is also negligable.

round(cor(V,U),5)
##          [,1]
## [1,] -0.00027

Same result can be seen by fitting a regression line through the data. The slope is almost zero.

Gaussian marginals does not imply joint gaussian

Suppose we have \(X \sim N \left( 0, 1\right)\) and \(W \sim U \left( \{ -1,1\}\right)\), i.e. \(W\) is uniform on \(-1, 1\), and \(W\) is independent of \(X\). Define \(Y = W X\). We will show that \(Y \sim N \left( 0, 1\right)\). Let \(y \in \mathbb{R}\) \[\begin{align*} F_Y(y) &= P(Y \leq y) = P(Y \leq y | W = -1)P(W = -1) + P(Y \leq y | W = 1)P(W = 1) \\ &= \frac{1}{2}P(X \leq y) + \frac{1}{2}P(-X\leq y) = P(X \leq y) = F_X(y), \end{align*}\] where we used the fact that X is symmetric around zero. Therefore we have \(X\) and \(Y\) which are both marginally \(N(0,1)\) distributed.

It is obvious, that they are not jointly gaussian. For example, it can be seen from the fact that for a two dimensional gaussian distribution \(Y | X = 1\) has to be gaussian, but in our case \((Y | X = 1) = W\) does not follow a gaussian distribution.

Whole situation can be clearly seen from a simple scatter plot.

W = 2*rbinom(n,1,1/2)-1
X = rnorm(n)
Y = W*X
plot(X,Y, col = 'darkgreen')
abline(v=1, lwd=2, lty=2, col = "red")
text(1.2, -3, labels=expression(X==1), col = "red", pos = 4)
points(c(1,1),c(-1,1), pch = 21, bg= 'red', cex = 2)